Lower Bounds for Bayes Error Estimation

ÐWe give a short proof of the following result. Let …X; Y † be any distribution on N f0; 1g, and let …X1; Y1†; . . . ; …Xn; Yn† be an i.i.d. sample drawn from this distribution. In discrimination, the Bayes error L ˆ infg Pfg…X† 6ˆ Y g is of crucial importance. Here we show that without further conditions on the distribution of …X; Y †, no rate-of-convergence results can be obtained. Let n…X1; Y1; . . . ; Xn; Yn† be an estimate of the Bayes error, and let f n…:†g be a sequence of such estimates. For any sequence fang of positive numbers converging to zero, a distribution of …X; Y † may be found such that E jL ÿ n…X1; Y1; . . . ; Xn; Yn†j f g an infinitely often. Index TermsÐDiscrimination, statistical pattern recognition, nonparametric estimation, Bayes error, lower bounds, rates of convergence. æ 1 INTRODUCTION THE pattern recognition problem may be formulated as follows: we are given n i.i.d. observations Dn ˆ f…X1; Y1†; . . . ; …Xn; Yn†g, drawn from the common unknown distribution of …X;Y † on R d f0; 1g. Given X, one must estimate Y as best as possible by a function gn…X† of X and the observations. The best one can hope for is to make an error equal to the Bayes error, L : Ln ˆ def Pfgn…X† 6ˆ Y jDng L ˆ def inf g:R !f0;1g Pfg…X† 6ˆ Y g: It is thus of great importance to be able to estimate L accurately, even before pattern recognition is attempted. Also, a comparison of estimates of Ln and L gives us an idea how much room is left for improvement. In a first group of methods, L is estimated by an estimate b Ln of the error probability Ln of some consistent classification rule gn. As such, this problem has been attempted by Fukunaga and Kessel [9], Chen and Fu [2], Fukunaga and Hummels [8], and Garnett and Yau [10], to cite just the early contributions. Concerning the error estimation of specific classification rules see Chapter 10 in McLachlan [11]. Clearly, if the estimate b Ln we use is consistent in the sense that b Ln ÿ Ln ! 0 with probability one as n!1, and the rule is strongly consistent, then b Ln ! L with probability one. In other words, we have a consistent estimate of the Bayes error probability. The problem is that even though for many classifiers, b Ln ÿ Ln can be guaranteed to converge to zero rapidly, regardless what the distribution of …X;Y † is (see Chapters 8, 23, 24, and 31 of Devroye et al. [7]), in view of Cover [3] and Devroye [4], the rate of convergence of Ln to L using such a method may be arbitrarily slow. Thus, we cannot expect a good performance for all distributions from such a method. The question thus is whether it is possible to come up with another method of estimating L (by n…X1; Y1; . . . ; Xn; Yn†) such that the difference n…X1; Y1; . . . ; Xn; Yn† ÿ L converges to zero rapidly for all distributions. Unfortunately, there is no method that guarantees a certain finite sample performance for all distributions. This disappointing fact is reflected in the following negative result (Theorem 8.5 of Devroye et al. [7]). Theorem 1. For every n, for any estimate n…X1; Y1; . . . ; Xn; Yn† of the Bayes error probability L , and for every > 0, there exists a distribution of …X;Y †, such that E j n…X1; Y1; . . . ; Xn; Yn† ÿ L j f g 1=4ÿ : The counterexamples in Theorem 1 vary with n, so it may still be possible that for every fixed distribution for …X;Y †, there exists a universal rate of convergence to zero for E j n…X1; Y1; . . . ; Xn; Yn† ÿ L j f g: The purpose of this note is to show that this too is impossible. We show the following: Theorem 2. For any sequence fang of positive numbers converging to zero, a distribution of …X;Y † on f1; 2; 3; . . .g f0; 1g may be found such that E j n…X1; Y1; . . . ; Xn; Yn† ÿ L j f g an